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This  research  has  investigated  several  topics  in  solving  unconstrained  minimizatioij 
systems  of  nonlinear  equations,  and  indefinite  linearly  constrained  minimization 
problems.  In  unconstrained  minimization,  a  new  projected  update  was  developed  and 
tested.  While  marginal  improvements  over  the  BFGS  were  found,  they  probably  do  no 
justify  the  use  of  the  new  update.  The  use  of  conic  models  for  unconstrained  min¬ 
imization  problems  when  analytic  or  finite  difference  derivatives  are  available 
was  also  investigated,  and  a  new  algorithm  was  developed  and  tested.  The  results 
show  reasonable  improvements  in  many  cases,  and  indicate  that  conic  algorithms  for 
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1.  Introduction 


Our  research  under  this  contract  has  mainly  been  concerned  with  the 

unconstrained  minimization  problem 

inin  f :  RM  -  R,  ... 

x,  Rn  ’ 

where  f  is  twice  continuously  differentiable.  At  times  we  have  also  con¬ 
sidered  the  closely  related  nonlinear  equations  problem 

given  F:  Rn  *  Rn,  find  x*  •  Rn  such  that  F(x*)  =  0.  (1.2) 

One  project  has  been  concerned  with  the  linearly  constained  minimization 
problem 

min  f:  Rn  *  R 
x  e  Rn 

subject  to  Cj(x)  i  0,  i  =  1 .  ra  (1.3) 

c.:  Rn  ►  R. 

The  main  topics  we  have  investigated  during  this  research  period  are: 

1.  Projected  updates  for  unconstrained  minimization. 

2.  Deriving  least  change  secant  updates  for  unconstrained  minimization  and 
nonlinear  equations. 

3.  Conic  models  for  unconstrained  minimization. 

4.  Tensor  models  for  systems  of  nonlinear  equations. 

5.  Algorithms  for  indefinite  linearly  constrained  minimization. 

A  minor  topic  investigated  was 

6.  Convergence  of  secant  approximations  to  the  correct  Jacobian  or  Hessian 
value. 

Our  findings  on  each  of  these  topics  are  discussed  briefly  in  Sections  2-7. 
Our  work  on  topics  1,  2,  and  6  is  essentially  complete,  while  topics  3,  4, 
and  5  were  started  towards  the  end  of  the  research  period,  and  are  the  main 
topics  we  are  currently  investigating  under  ARO  contract  DAAG2<)-81-k-01Rl , 
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which  started  on  June  lb,  1981. 

A  list  of  publications  and  technical  reports  durinq  the  research  period 
is  given  in  Section  8.  However,  due  to  our  co-authorship  of  a  research 
book  on  unconstrained  minimization  and  nonlinear  equations  with  J.  Dennis, 
which  will  be  completed  in  Fall  1981,  there  are  several  papers  reporting  our 
research  under  this  contract  that  have  not  been  completed.  They  are  listed 
separately  in  Section  8.  The  findinqs  of  two  of  these  papers  are  partially 
reported  in  two  of  the  M.S.  theses  that  are  listed  in  Section  9.  Most  of 
these  research  results  have  been  reported  at  major  scientific  meetings  in 
our  field;  we  list  these  presentations  in  Section  10. 

2.  Projected  updates  for  unconstrained  minimization 

A  new  technique  for  deriving  the  BFGS  update,  developed  by  Dennis,  was 
extended  by  us  to  derive  a  new  projected  BFGS  update.  (All  this  research  is 
reported  in  Dennis  and  Schnabel  [4].)  This  update  is  the  same  as  Davidon's 
[1]  projected  update  on  quadratic  functions,  but  is  different  in  general.  Our 
research  assistant,  M.  LaRue,  conducted  an  extensive  series  of  tests  com¬ 
paring  an  algorithm  using  the  new  update  to  the  same  algorithm  using  the 
standard  BFGS.  Over  the  entire  More-Garbow-ili  1  lstrom  [6]  test  set,  the 
improvements  were  small;  while  on  the  average  the  new  update  did  slightly 
better,  on  a  fair  number  of  problems  this  wasn't  the  case.  We  still  plan 
to  write  up  the  results  of  this  research,  but  in  view  of  the  added  com¬ 
plexity  of  the  new  update,  the  small  improvements  do  not  seem  to  warrant 
its  use  in  general  purpose  software. 

To  us,  the  most  important  contribution  of  the  computational  research 
was  the  techniques  and  intuition  we  developed  about  incorporating  information 
from  past  iterates  into  our  algorithms.  Basically,  we  learned  that  at 
iteration  k,  it  did  not  make  sense  to  use  information  from  iterate  k-i 
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unless  the  angle  between  x^  -  .  and  the  subspace  spanned  by 

{ x |^ - x k  j  |  j=l , . . .  ,i-l)  was  large,  say  4b  .  Secondly,  it  seemed  that  even 
though  the  theory  permitted  using  information  from  up  to  n  past  points,  an 
upperbound  of  about  /n  was  optimal  in  practice.  These  two  observations 
were  very  important  to  our  subsequent  work  on  conic  and  tensor  models. 


3.  Deriving  least  change  secant  updates  for  unconstrained  minimization  and 
nonlinear  equations 

We  have  continued  our  work  on  this  topic  that  was  started  in  Dennis  and 

Schnabel  [2].  A  paper  currently  in  draft  form  by  Buckley  and  Schnabel  contains 

different  techniques  for  deriving  the  updates  in  [3],  and  uses  these  techniques 

to  derive  a  new  weighted  least  change  sparse  symmetric  secant  update.  Another 

paper  in  draft  form  by  Schnabel  and  Toint  derives  the  weighted  symmetric 

sparse  projection  operator,  the  solution  to 

min  ljW'1/2(B  -  A)  W"1/2j|. 

B  ,  Rnxn 

subject  to  B  symmetric  and  obeying  a  specific  sparsity  pattern 
where  A  e  Rn  n  is  symmetric  and  W  =  1  +  a  matrix  of  rank  k,  k  •  n.  This 
operator  is  shown  in  [3]  to  be  a  key  to  deriving  weighted  least  change  sparse 
secant  updates.  Unfortunately,  the  results  of  Schnabel  and  Toint  show  that 
for  k  >  1,  computing  B  is  computationally  unattractive. 

4.  Conic  models  for  unconstrained  minimization 

This  research  is  based  on  the  seminal  work  of  Davidon  [2]  and  the  sub¬ 
sequent  work  of  Sorensen  [8],  and  is  mostly  reported  in  the  M.S.  thesis 

by  Stordahl  (see  Section  9).  It  concerns  using  the  conic  model 

T  I  T 

m(x+d)  =  f  +  g  d  +  2  d  Ad  (4.1) 

l+pTd  ( l+pTd) 2 


T 


where  f  c  R,  x,  d,  g,  p c  Rn,  A  f  Rn  ,  in  place  of  the  standard  quadratic  model 
in  minimization  algorithms.  Davidon's  and  Sorensen’s  work  concerned  using 
(4.1)  in  an  algorithm  where  only  first  derivatives  were  available;  our  re¬ 
search  has  concerned  the  case  where  analytic  or  finite  difference  second 
derivatives  are  also  available.  We  show  how  to  use  (4.1)  to  interpolate 
f(x) ,  Vf(x)  and  V  f(x)  at  the  current  iterate  x^,  as  well  as  f(x)  at  up  to 
n  past  iterates  ^ .  In  practice,  at  most  ft  past  iterates  are  used  at 
any  iteration.  Stordahl  tested  an  algorithm  making  very  simple  use  of  this 
model  on  the  test  set  from  [6],  and  found  average  improvements  of  15-30 
in  iterations  and  function  evaluations  over  the  corresponding  algorithm 
using  a  quadratic  model,  although  in  some  cases  the  new  algorithm  performed 
worse.  These  results  have  been  presented  at  two  international  conferences 
(see  Section  10)  and  will  be  reported  in  a  forthcoming  paper. 

5.  Tensor  models  for  systems  of  nonlinear  equations 

This  research,  motivated  by  our  work  on  conic  models  for  minimization. 


attempts  to  use  the  model 


l  u 

M(x+d)  =  F  +  Jd+^-Td 


(5.1) 


where  x,d,FcRn,  J  e  Rn  n,  T  r"  n  n,  in  solving  the  nonlinear  eouations  problem 
(1.2).  Using  the  full  term  T  is  out  of  the  question,  since  it  would  require 

3 

n  storage,  and  the  solution  of  a  system  of  n  quadratics  in  n  unknowns  to 
find  the  root  of  the  model  at  each  iteration.  Our  research  has  shown  how 
to  efficiently  use  a  small  portion  of  T  to  improve  the  performance  of  non¬ 
linear  equations  algorithms,  especially  in  problems  where  the  Jacobian  at 
the  solution  is  singular,  without  significantly  increasing  the  storage 
required  or  the  arithmetic  cost  per  iteration.  At  the  k^  iteration,  we 
use  (5.1)  to  interpolate  F^),  F  (x^),  and  Hx^.)  at  m  <  yfn  past  iterates. 

The  first  two  conditions  require  F  =  F( x^ )  and  0  =  F'fx^);  we  show  that 


-5- 

the  smallest  T  (in  the  Frobenius  norm)  that  accomplishes  the  last  condition 
is  a  tensor  of  rank  m  that  is  stored  using  2 m  n-vectors,  that  is,  at  most 
2n  7  storage.  We  also  show  that  a  root  of  M(x+d)  can  then  be  found  by 
solving  a  system  of  m  quadratics  in  m  unknowns,  which  is  inexpensive 
compared  to  the  one  n  >  n  linear  system  that  is  solved  at  each  iteration  of 
any  nonlinear  equations  algorithm.  A  preliminary  implementation  of  this 
algorithm  is  reported  in  the  M.S.  thesis  by  Trank  (see  Section  9);  it 
contains  several  additional  important  features.  So  far  the  algorithm  has 
performed  as  well  or  better  than  the  corresponding  algorithm  using  a 
standard  linear  model  on  virtually  every  test  problem,  and  on  many  singular 
problems  the  improvements  were  quite  large.  This  research  is  still  in  a 
preliminary  stage  and  is  being  continued  on  our  new  ARO  grant.  It  was 
reported  at  a  recent  international  meeting  (see  Section  10),  where  several 
knowledgeable  people  said  that  they  thought  it  had  very  good  promise. 

6.  Algorithms  for  indefinite  linearly  constrained  minimization 

Our  research  assistant  during  the  last  year  of  this  contract,  J.  Shultz, 
has  been  conducting  his  Ph.D.  research  on  this  topic.  It  is  motivated  by  the 
algorithm  we  introduced  in  [7]  for  finding  whether  a  system  of  linear  and 
nonlinear  inequality  constraints  has  a  feasible  point.  This  algorithm  is 
of  considerable  practical  interest,  for  example  as  a  Phase  I  procedure  for 
G.R.G.  codes  for  constrained  minimization,  but  to  implement  it  successfully 
requires  a  new  algorithm  for  indefinite  linearly  constrained  minimization. 

Our  research  with  Shultz  has  focused  on  two  topics:  strategies  for  dropping 
active  constraints  when  the  reduced  Hessian  is  indefinite,  and  step  direction 
strategic  when  the  reduced  Hessian  is  indefinite.  We  believe  we  have 
satisfactorily  solved  both  problems,  and  at  this  time,  Shultz  is  testing  t ho 
new  algorithm.  The  results  will  be  reported  in  a  joint  paper  and  in  his 
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7.  Convergence  of  secant  approximations  to  the  correct  Jacobian  or  Hessian 
value 

At  the  Fall  1980  SIAM  meeting  in  Houston,  we  presented  a  series  of  new 
examples  showing  that  the  sequence  of  Jacobian  or  Hessian  approximations 
generated  by  secant  algorithms  (Broyden's  method,  BFGS,  DFP,  PSB)  do  not 
necessarily  converge  to  the  Jacobian  or  Hessian  matrix  at  the  solution,  even 
when  the  iterates  converge  to  the  solution  q-superl inearly.  These  results 
show  that  if  the  Jacobian  is  constant  in  some  of  its  rows,  then  Broyden's 
method  (or  any  similar  rank  one  update  method)  will  almost  always  generate 
a  sequence  of  approximations  that  converges  to  incorrect  values  in  the  other, 
nonlinear  rows.  This  behavior  is  replicated  in  practice  and  similar  behavior 
is  sometimes  noted  when  some  component  functions  are  far  more  nonlinear  than 
others.  For  minimization,  we  show  that  the  same  behavior  can  occur  if  the 
Hessian  is  constant  in  some  rows  (and  columns),  but  if  the  BFGS,  DFP  or  any 
other  update  from  the  Broyden  class  is  used,  it  seems  far  less  likely  to  occur 
However,  the  PSB  updates  converge  to  an  incorrect  derivative  value  when  the 
BFGS  and  Broyden  class  converge  correctly,  and  so  this  isolates  another  reason 
why  unweighted  symmetric  secant  updates  are  inferior  to  weighted  symmetric 
secant  updates  for  minimization.  This  research  is  partially  reported  in 
[5],  and  will  be  a  forthcoming  paper. 

8.  Publications  and  technical  reports  produced 

1.  "Determining  feasibility  of  a  set  of  nonlinear  inequality  constraints," 
to  appear  in  Math.  Prog.  Study  on  Constrained  Optimization. 

2.  "A  new  derivation  of  symmetric  positive  definite  secant  updates,"  in 
Nonlinear  Programming  4,  0.  L.  Mangasarian,  R.  R.  Meyer,  S.  M.  Robinson, 
eds..  Academic  Press,  N.  Y.,  1981  (with  J.  E.  Dennis,  Jr.). 

"Comments  on  evaluating  algorithms  and  codes  for  mathematical  programming, 
to  appear  in  Proc'  ings  of  the  Boulder  COAL  conference. 


3. 
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4.  "Unconstrained  optimization  in  1981,"  to  appear  in  the  proceedings  of  the 

NATO  ARI  on  Nonlinear  Optimization,  M.  J.  D.  Powell, ed..  Academic  Press,  198?. 
Unconstrained  Optimization  and  Nonlinear  Equations,  Prentice-Hall, 

New  Jersey,  1982  (with  J.  E.  Dennis,  Jr.). 

The  following  papers  currently  are  in  nearly-final  draft  form: 

"Long  vectors  for  secant  updates"  (with  A.  Buckley) 

"Forcing  sparsity  by  projecting  with  respect  to  a  non-diagonal ly 
weighted  Frobenius  norm"  (with  Ph.  L.  Toint). 

The  following  are  some  additional  papers  that  will  be  forthcoming  reporting 
work  performed  under  this  contract: 

"Projected  secant  updates  for  unconstrained  minimization" 

"On  the  convergence  of  Jacobian  and  Hessian  approximations  to  the  correct 
derivative  value" 

"Conic  models  using  second  derivatives  for  unconstrained  minimization" 

"Tensor  models  for  solving  systems  of  nonlinear  equations"  (with  P.D.  Frank) 

"A  modular  system  of  algorithms  for  unconstrained  minimization" 

(with  B.  E.  Weiss  and  J.  E.  Koontz) 

9.  List  of  scientific  personnel 

M.  LaRue,  research  assistant,  June-July  1979,  Jan. -July  1980 

J.  Shultz,  research  assistant.  Sept.  1980-May  1981  (Ph.D.  expected,  1982) 

Masters  theses  supervised  in  conjunction  with  the  research  for  this  contract: 

B.  E.  Weiss,  "A  modular  software  package  for  solving  unconstrained  non¬ 
linear  optimization  problems,"  1980 

K.  A.  Stordahl,  "Unconstrained  minimization  using  conic  models  and  exact 

second  derivatives,"  1980 

P.  D.  Frank,  "A  second-order  local  model  for  solution  of  systems  of  nonlinear 


equations,"  1981 
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10.  Presentations  of  research  conducted  under  this  contract  at  scientific 
meeti ngs 

1.  "Determining  feasibility  of  a  system  of  nonlinear  inequality  constraints” 

Tenth  International  Math  Programing  Symposium,  Montreal,  Aug.  1979  and 
ORSA-TIMS  fleeting,  Milwaukee,  October  1979. 

2.  "Consideration  of  scaling  in  unconstrained  optimization  algorithms" 

SIAM  Meeting,  Denver,  Nov.  1979. 

3.  "A  derivation  of  the  BFGS  from  the  Broyden  update  and  some  consequences" 
by  J.  Dennis,  coauthor,  at  nonlinear  Programming  Symposium  IV, 

Madison,  July  1980. 

4.  "On  the  convergence  of  secant  approximations  to  the  correct  derivative 
value"  SIAM  Meeting,  Houston,  Nov.  1980. 

5.  "Unconstrained  minimization  using  conic  models  and  exact  second  derivatives" 
Mathematical  Programming  Conference,  Oberwohl fach ,  W.  Germany,  Jan.  1981. 

6.  "Unconstrained  optimization  in  1981"  and  "Nonstandard  models  for  unconstrained 
optimization  and  nonlinear  equations"  NATO  ARI  on  Nonlinear  Optimization, 
Cambridge,  England,  July  1981. 
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